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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  conducts 
research  to  support  Army  personnel  and  training  goals.  In  recognition  of  the  changes  emerging 
with  the  Army’s  transformation,  ARI  developed  a  research  program  to  identify,  describe,  and 
address  future  personnel  requirements.  This  report  describes  an  aspect  of  an  ongoing  ARI 
project,  called  Select21,  concerned  with  future  enlisted  Soldiers. 

The  objective  of  this  project  is  to  provide  personnel  tests  for  selecting  and  assigning 
entry-level  Soldiers  to  future  jobs.  This  report  is  part  of  a  series  of  Product  Reports  summarizing 
interim  outcomes  of  this  ongoing  project. 

CD  Future  job  clusters  -  February  2005  (Research  Product  2005-04) 

I  I  Future  Army-wide  Soldier  performance  requirements  -  March  2005  (Research 
Product  2005-05) 

CD  Soldier  job  performance  measurement  tools  -  August  2005  (Research  Product 
2005-07) 

I7|  Select21  experimental  selection  and  classification  instruments  -  October  2005 

CD  Select21  validation  results  and  recommendations 

This  report  describes  the  experimental  selection  and  classification  instruments.  In  the 
Select21  project,  Soldiers’  scores  on  these  instruments  will  be  linked  to  their  scores  on  job 
performance  measures  to  evaluate  how  well  these  instruments  might  forecast  future  job 
performance.  Instruments  that  show  promise  for  predicting  job  performance  could  supplement 
the  Department  of  Defense’s  current  selection  and  classification  test  battery  (i.e.,  the  Armed 
Services  Vocational  Aptitude  Battery  [AS  VAB])  for  the  Army  in  the  future. 

Project  Select21  is  being  conducted  with  support  from  the  Army  G-l,  Deputy  Chief  of 
Staff  for  Personnel,  and  from  the  Army  Training  and  Doctrine  Command  (TRADOC).  ARI  has 
briefed  these  sponsors,  as  well  as  representatives  of  other  offices  to  include  the  Army  Accessions 
Command,  Human  Resources  Command,  and  the  Army  G-3,  Deputy  Chief  of  Staff  for 
Operations.  Research  sponsors  have  provided  the  support  and  guidance  needed  for  the  success  of 
the  research. 


MICHELLE  SAMS 
Technical  Director 
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What  is  Project  Select21? 

The  U.S.  Army  has  undertaken  far-reaching  changes  to  transform  the  current  force  into  one 
that  is  more  responsive,  deployable,  agile,  versatile,  and  lethal  while  being  fully  survivable  and 
sustainable  under  all  conditions.  New  Predictors  for  Selecting  and  Assigning  Future  Force  Soldiers 
(SelectU)  is  an  Army  research  project  focused  on  the  personnel  system  by  which  the  Army  selects 
entry-level  Soldiers  and  assigns  these  Soldiers  to  jobs.  Select21  is  designed  to  help  ensure  that 
through  this  system,  the  Army  will  acquire  new  Soldiers  with  the  knowledges,  skills,  and  attributes 
(KSAs)  needed  for  performing  the  types  of  tasks  emerging  as  part  of  the  transformation.  More 
specifically,  the  objectives  of  Select21  are  to  develop  measures  of  these  KSAs  and  to  evaluate  their 
potential  for  integration  into  the  Army’s  personnel  acquisition  system. 

Figure  1  shows  a  schematic  overview  of  the  Select21  project.  To  understand  how  well  the 
experimental  selection  and  classification  tests  work,  we  are  comparing  how  well  Soldiers  perform  on 
these  tests  with  how  well  these  same  Soldiers  perform  their  jobs.  Select21  will  use  a  “concurrent 
validation”  design.  With  this  design,  researchers  are  administering  the  experimental  tests  to  a  sample 
of  Soldiers  with  1 8  to  36  months  time  in  service  and,  at  the  same  time,  assessing  the  Soldiers’  job 
performance  by  using  specially  designed  indicators,  called  criterion  measures. 


Jan  2002  Jan  2003 


Jan  2004 


Jan  2005 


Figure  1.  Schematic  of  the  Select21  project  plan. 

What  is  this  report  about? 

This  report  describes  the  experimental  selection  and  classification  instruments  developed 
in  Select21.  The  main  purpose  is  to  acquaint  Army  readers  with  the  purpose,  content,  and  format 
of  these  instruments  by  presenting  descriptions,  examples,  and  a  brief  overview  of  the 
development  process. 
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Experimental  Selection  and  Classification  Instruments 


A  fundamental  goal  of  the  Select21  project  is  to  develop  experimental  selection  and 
classification  measures  that  will  (a)  predict  performance  for  entry-level  Soldiers  in  the  Future 
Force  and  (b)  add  “incremental  validity”  over  the  current  system  as  embodied  by  the  Armed 
Services  Aptitude  Battery  (ASVAB).  “Incremental  validity”  means  that  a  new  measure  can 
predict  performance  over  and  above  what  we  can  achieve  with  the  already  available  ASVAB. 
This  is  a  difficult  goal,  since  there  is  a  great  deal  of  research  documenting  the  strong  predictive 
validity  of  the  ASVAB.  Toward  those  ends,  the  Select21’s  experimental  selection  and 
classification  instruments  were  designed  to  measure  KS  As  that  are  not  already  measured  by  the 
ASVAB  but  are  expected  to  be  important  in  the  Future  Force. 

The  experimental  selection  and  classification  instruments  include  the  following: 

•  Rational  Biodata  Inventory  (RBI) 

•  Work  Suitability  Inventory  (WSI) 

•  Psychomotor  Tests 

•  Predictor  Situational  Judgment  Test  (PSJT) 

•  Record  of  Pre-Enlistment  Training  and  Experiences  (REPETE) 

•  Work  Values  Inventory  (WVI) 

•  Work  Preferences  Survey  (WPS) 

•  Career  Exploration  Program  (CEP)  Interest  Inventory 

•  Army  Beliefs  Survey  (ABS) 

•  Pre-Service  Expectations  Survey  (PSES) 

•  Army  Work  Knowledge  Survey  (AWKS) 

The  experimental  selection  and  classification  instruments  have  been  finalized  in 
preparation  for  collecting  concurrent  validation  data  in  FY05.  For  test  security  reasons  some  of 
the  measures  cannot  be  released  in  their  entirety.  The  following  pages  provide  descriptions  and 
examples  for  each  of  the  instruments.  Following  this  presentation  is  a  brief  description  of  how 
the  instruments  were  developed. 
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Rational  Biodata  Inventory  (RBI) 


The  Rational  Biodata  Inventory  (RBI)  is  designed  to  measure  motivational/temperament 
characteristics  identified  as  important  to  entry-level  Soldier  performance  in  the  Army’s  future 
force.  The  RBI  incorporates  biodata  scales  from  several  existing  Army  tests  designed  to  predict 
job  performance  and  counterproductive  tendencies.  These  existing  tests  are  used,  for  example, 
for  selecting  Special  Forces  Soldiers. 

The  RBI  consists  of  90  “biodata”  items  and  takes  30  minutes  to  administer.  Biodata  items 
ask  about  past  behavior  and  reactions  to  previous  life  events  (e.g.,  “to  what  extent  have  you 
enjoyed  thinking  about  the  plusses  and  minuses  of  alternative  approaches  to  solving  a  problem”). 
The  RBI  items  are  written  specifically  to  tap  important  motivational  and  temperament 
characteristics  such  as  the  following: 

•  Achievement  Orientation:  Working  hard  and  giving  one’s  best  effort 

•  Cognitive  Flexibility:  Willingness  to  try  new  ways  of  getting  work  done. 
Embracing  innovation  and  change.  Comfort  with  uncertainty  and  ambiguity. 

•  Peer  Leadership:  Comfort  with  taking  the  initiative.  Willing  to  make  tough 
decisions  and  accept  responsibility  for  the  group’s  performance. 

•  Cultural  Tolerance:  Willingness  to  accept  and  work  closely  with  those  from  other 
cultural/religious/ethnic  backgrounds. 

Following  is  an  example  of  a  rational  biodata  item: 


How  often  have  you  put  off  doing  a  chore  that  you  could  have  taken  care  of  right  away? 

A.  Very  often 

B.  Often 

C.  Sometimes 

D.  Seldom 

E.  Never 
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Work  Suitability  Inventory  (WSI) 


The  Work  Suitability  Inventory  (WSI)  uses  a  self-report  format  to  determine  the  types  of 
work  that  respondents  believe  they  would  perform  best.  The  WSI  differs  from  many  other  tests 
in  that  it  asks  respondents  to  sort  statements  rather  than  answer  questions.  Specifically,  this 
computerized  measure  asks  respondents  to  sort  16  statements  describing  different  types  of  work 
requirements.  Figure  2  shows  an  example  of  two  of  the  statements  that  appear  on  the  WSI. 

Each  statement  appears  on  its  own  rectangular  block,  or  “card.”  Respondents  must  rank 
the  16  statements  in  terms  of  how  successful  they  think  they  would  be  at  performing  each  type  of 
work  described — the  highest  ranked  statement  should  describe  work  that  respondents  think  they 
would  be  most  successful  at,  and  the  lowest  ranked  statement  should  describe  work  that 
respondents  think  they  would  be  least  successful  at.  To  rank  the  computerized  “type  of  work” 
cards,  respondents  use  the  mouse  to  position  the  cards  on  spaces  labeled  1  (Most  Successful) 
through  16  (Least  Successful)  (see  Figure  2).  Respondents  have  12  minutes  to  complete  the  WSI. 

An  important  feature  of  the  WSI  lies  in  its  scoring  approaches.  The  Army  can  score  the 
WSI  differently  for  each  outcome  of  interest  (e.g.,  job  performance,  attrition,  person-Army  fit). 
Unlike  conventional  tests  having  correct  answers  or  a  single  set  of  keyed  answers,  no  single 
ordering  of  the  16  cards  will  likely  predict  all  outcomes  of  interest — a  ranking  that  yields  a  high 
score  on  one  outcome  may  well  yield  a  low  score  on  other  outcomes.  Therefore,  respondents’ 
attempts  to  rank  the  statements  the  way  they  think  the  Army  would  like  (rather  than  ranking 
them  in  the  way  that  best  describes  themselves)  will  not  be  successful.  As  such,  the  WSI  should 
be  resistant  to  response  distortion,  which  happens  when  a  respondent  tries  to  answer  questions  to 
give  the  most  favorable  impression  possible  rather  than  to  be  most  accurate. 
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Most 

Successful 


Wbrl:  that  requires... 
being  open  to  change 
(positive  and  negative) 
and  a  lot  of  variety. 


Least 

Successful 


Figure  2.  Screenshot  of partial  sorting  of  statements  from  the  Work  Suitability  Inventory. 
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Psychomotor  Tests 


There  is  good  reason  to  expect  that  tests  of  psychomotor  ability  might  improve  upon  the 
validity  of  the  ASVAB  for  predicting  certain  aspects  of  future  job  performance  in  entry-level 
Army  MOS.  Prior  research  has  shown  psychomotor  tests  to  be  good  predictors  of  gunnery 
performance  and  certain  other  job  performance  criteria.  Additionally,  the  Select21  job  analysis  data 
suggested  that  psychomotor  skills  were  more  important  for  performance  in  Close  Combat  jobs  and 
therefore  might  be  useful  for  classification  purposes. 

The  Select21  psychomotor  tests — Target  Tracking  and  Target  Shoot — were  adapted  from 
an  earlier  Army  research  project.  They  are  designed  to  measure  a  person’s  ability  to  adjust  or 
position  a  machine  control  mechanism  precisely.  For  both  tests,  the  respondent  uses  a  joystick  to 
track  a  target.  Both  tests  are  administered  on  an  IBM  Thinkpad  computer,  and  together  they  take 
approximately  25  minutes  to  administer. 


Target  Tracking  Test 

On  each  item  of  the  test,  a  path  consisting  of  vertical  and  horizontal  line  segments  is 
presented.  A  sample  item  is  shown  in  Figure  3.  A  target  box  appears  at  the  beginning  of  the  path. 
As  the  item  begins,  the  target  starts  to  move  along  the  path  at  a  constant  rate  of  speed.  The 
respondent’s  task  is  to  keep  the  crosshair  centered  within  the  target  at  all  times  using  a  joystick 
to  control  movement.  The  respondent’s  score  on  this  test  indicates  how  accurately  he  or  she 
tracked  the  target  (i.e.,  the  average  distance  from  the  center  of  the  crosshair  to  the  center  of  target 
across  all  items).  The  Target  Tracking  Test  contains  three  practice  and  nine  scored  items. 


Target  Shoot  Test 

At  the  beginning  of  an  item  on  this  test,  a  crosshair  appears  in  the  center  of  the  screen  and 
a  target  box  appears  at  some  other  location  on  the  screen,  as  shown  at  the  top  of  Figure  4.  The 
target  then  begins  to  move  around  the  screen  in  an  unpredictable  manner,  frequently  changing 
direction.  The  respondent  controls  movement  of  the  crosshair  using  a  joystick.  The  respondent’s 
task  is  to  move  the  crosshair  into  the  center  of  the  target  and  press  a  red  button  on  the  joystick  to 
“fire”  at  the  target.  If  successful,  a  “hit”  is  displayed  as  shown  in  the  lower  half  of  Figure  4.  The 
respondent  must  do  this  before  the  time  expires  on  each  item.  The  respondent  receives  two 
scores  on  this  test — a  speed  score  and  an  accuracy  score.  The  speed  score  is  the  average  time 
elapsed  from  the  beginning  of  the  item  until  the  respondent  fired  at  the  target.  The  accuracy 
score  is  the  average  distance  from  the  center  of  the  crosshair  to  the  center  of  the  target  at  the  time 
the  respondent  fired  at  the  target.  The  Target  Shoot  Test  contains  3  practice  and  52  scored  items. 
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Figure  3.  Example  Target  Tracking  Test  item. 
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Predictor  Situational  Judgment  Test  (PSJT) 


Situational  Judgment  Tests  have  become  very  popular  in  recent  years  because  they  have 
shown  good  validity  for  predicting  job  performance  and  incremental  validity  over  general 
cognitive  ability  measures  like  the  ASVAB.  The  purpose  of  Select21  ’s  PSJT  is  to  predict  future 
first-term  Soldier  job  performance  by  simulating  situations  an  individual  Soldier  faced  prior  to 
enlistment  as  a  Soldier.  Each  item  consists  of  a  description  of  a  problem  situation  and  a  list  of  four 
alternative  actions  that  a  person  might  take  in  that  situation.  A  sample  item  is  shown  below. 


Instructions:  You  are  to  rate  the  effectiveness  of  each  action  using  a  1-7  rating  scale,  where  7  is 
highly  effective  and  1  is  completely  ineffective.  Imagine  that  you  are  in  the  situation;  use  the 
scale  to  indicate  how  effective  or  ineffective  you  believe  each  action  to  be. 

Sample  Item:  You  are  a  student  working  on  a  group  project  with  your  classmates.  You  are 
approaching  your  deadline  and  the  project  is  not  complete.  One  member  of  the  group  offers 
to  take  the  project  home  and  complete  it  over  the  weekend.  How  effective  is  each  of  the 
following  actions  you  could  take? 

a.  Offer  to  take  the  project  home  and  finish  it  instead  of  the  other  student  taking 
it  home. 

b.  Plan  a  day  for  all  the  group  members  to  get  together  and  work  on  the  project. 

c.  Ask  the  student  if  he/she  needs  help. 

d.  Let  the  student  take  it  home  and  complete  the  project. 


The  PSJT  measures  a  respondent’s  judgment  in  civilian  situations  that  are  similar  to 
military  situations  experienced  by  Soldiers  during  their  first  few  months  in  the  Army.  During  the 
development  of  the  test,  non-commissioned  officers  (NCOs)  were  asked  to  write  actual  situations 
that  Soldiers  have  experienced  during  their  first  few  months  in  the  Army.  For  each  of  these 
military  situations,  Soldiers  in  training  and  civilians  wrote  civilian  situations  that  paralleled  the 
military  situation.  In  most  cases,  the  situation  involved  either  a  student  at  a  civilian  school  or  an 
employee  at  a  civilian  organization.  The  test  development  process  had  several  rounds  of  writing, 
editing,  pilot  testing,  and  analysis. 

PSJT  Items  were  written  to  specifically  target  the  following  areas: 

•  Adaptability  to  Changing  Conditions 

•  Effective  Self-Management 

•  Effective  Self-Directed  Learning 

•  Relating  to  and  Supporting  Peers 

•  Teamwork 

The  concurrent  validation  version  of  the  PSJT  contains  26  items  and  takes  about  45 
minutes  to  administer. 
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Record  of  Pre-Enlistment  Training  and  Experiences  (REPETE) 


Historically,  the  Army  has  taken  the  burden  of  training  all  required  entry-level  job  skills 
for  its  enlisted  personnel.  It  stands  to  reason  that  recognizing  prior  related  training  and/or 
experience  could  benefit  the  Army  by  reducing  training  requirements  (or  at  least  helping  to 
ensure  success  in  training)  and  benefit  applicants  by  enhancing  their  enlistment  options  (in  terms 
of  job  choices  and/or  enlistment  bonuses). 

The  development  of  the  Record  of  Pre-Enlistment  Training  and  Experience  (REPETE) 
was  a  demonstration  effort — one  designed  to  illustrate  what  kind  of  instrument  might  be 
developed  and  how  it  might  be  used.  The  demonstration  effort  focused  on  computer  skills 
because  computer  skills  are  somewhat  important  for  all  MOS,  particularly  important  for  some 
MOS,  and  not  addressed  directly  on  the  ASVAB. 

The  REPETE  is  a  self-report  measure  designed  to  determine  the  type  of  training  and 
experience  that  entry-level  Soldiers  currently  bring  with  them  to  the  Army.  It  focuses  on  the 
following  10  computer  skill  categories: 


1.  Word  Processing/Desktop  Publishing  Software — Create,  manipulate,  format,  and  print  documents. 

2.  Spreadsheet  Software — Record,  format,  sort,  analyze,  and  graph  information. 

3.  Database  Software — Create,  query,  organize,  analyze,  graph,  and  report  databases. 

4.  Presentation/  Graphics  Software—Create  presentation-quality  slides  or  graphics. 

5.  Internet  Usage  and  Information  Search— Send,  receive,  and  save  email;  search  the  Internet. 

6.  Basic  Hardware  and  Operating  Systems— Manage  own  PC  files  and  folders  using  the  operating 
system  and  hardware. 

7.  Networking  and  Computer  Service  Support — Install,  initialize,  configure,  and  manage  network 
software. 

8.  Computer  Programming  Principles — Develop  algorithms,  select  programming  languages,  design 
program,  use  assembly  language,  and  develop  documentation. 

9.  Basic  Web  Programming— Develop  web  pages  using  HTML  and  Javascript. 

10.  Object-Oriented  Programming  Concepts — Create  Object-Oriented  Modeling  using  UML  notation; 
use  C++,  Java,  or  Visual  Basic. 


The  REPETE  has  three  parts,  each  with  multiple  items.  In  the  first  part,  respondents  are 
asked  to  list  courses  they  have  taken  related  to  computer  skills  and  to  indicate  which  of  the  10 
computer  skills  categories  were  addressed  in  each  course.  The  second  part  is  structured  similarly 
but  asks  about  certifications.  The  last  part  asks  respondents  to  rate  themselves  on  each  of  10 
computer  skill  categories  using  a  5-point  level-of-mastery  rating  scale. 

Because  the  REPETE  is  not  a  predictor  in  the  traditional  sense  and  because  it  would 
require  Soldiers  to  try  to  recall  their  pre-enlistment  status  with  regard  to  the  questions,  it  will  not 
be  included  in  the  concurrent  validation  effort.  With  additional  research,  however,  the  REPETE 
could  be  could  be  expanded  and  used  to  give  credit  for  a  broad  range  of  pre-enlistment  skills  and 
experience.  Such  a  tool  would  supplement  the  current  Army  Civilian  Acquired  Skills  Program 
(ACASP),  which  already  offers  enlistment  bonuses,  accelerated  promotions,  and/or  abbreviated 
training  for  those  enlistees  who  meet  certain  MOS-specific  skills  or  certification  criteria. 
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Work  Values  Inventory  (WVI) 


People  tend  to  value  different  things  when  it  comes  to  work  (e.g.,  physical  activity, 
autonomy).  Similarly,  organizations,  such  as  the  Army,  tend  to  vary  in  the  degree  to  which  they 
offer  characteristics  of  work  that  individuals’  value.  The  purpose  of  the  Work  Values  Inventory 
is  to  assess  respondents’  preferences  for  various  characteristics  of  work  (e.g.,  On  my  ideal  job,  I 
would  make  decisions  on  my  own).  It  is  based  on  the  premise  that  not  all  respondents  value 
characteristics  of  work  offered  by  the  Army,  and  that  a  disconnect  between  what  respondents  value 
and  what  the  Army  offers  can  lead  to  undesirable  outcomes.  Specifically,  past  research  suggests 
that  individuals  who  value  work  characteristics  that  their  organization  provides  are  more  satisfied 
and  more  likely  to  remain  in  the  organization  than  those  who  value  characteristics  not  provided 
by  their  organization. 

The  WVI  is  a  computerized  assessment  that  yields  an  ordering  of  28  work  characteristics 
in  terms  how  important  they  would  be  in  their  ideal  job,  and  distinguishes  between  important  and 
unimportant  characteristics  (in  an  absolute  sense).  To  make  the  rank  ordering  activity  easier, 
respondents  are  first  asked  to  sort  the  28  work  characteristics  into  four  categories  of  varying 
importance  (A-D).  For  example,  respondents  place  their  seven  most  important  work 
characteristics  in  Category  A  and  their  seven  least  important  work  characteristics  in  Category  D. 
Respondents  then  rank  order  the  importance  of  the  work  characteristics  within  each  category. 
After  completing  their  rankings  within  each  category,  respondents  are  presented  with  the  full  list 
of  28  work  characteristics  in  the  order  they  ranked  them.  Respondents  are  asked  to  review  the  list 
and  to  make  a  line  through  it  such  that  the  work  characteristics  important  for  their  ideal  job  are 
above  the  line  and  the  characteristics  below  the  line  are  unimportant  to  have  on  their  ideal  job. 

The  WVI  takes  about  25  minutes  to  administer.  Once  a  respondent  completes  the  WVI, 
the  Army  can  assess  the  similarity  of  his  or  her  preferences  for  various  work  characteristics  with 
what  the  Army  actually  offers. 


11 


Work  Preferences  Survey  (WPS) 


Occupations  can  be  categorized  into  six  types:  Realistic,  Investigative,  Artistic,  Social, 
Enterprising,  and  Conventional.  Individuals  vary  in  the  degree  to  which  they  are  interested  in 
these  types  of  occupations.  Research  has  shown  that  people  are  drawn  to  occupations  that  they 
think  will  satisfy  their  interests.  For  example,  people  who  are  creative  and  expressive  tend  to 
seek  out  Artistic  occupations.  Studies  have  also  found  that  a  good  match  between  an  individual’s 
interests  and  the  interests  that  his  or  her  chosen  occupation  supports  can  lead  to  a  variety  of 
positive  outcomes  for  both  the  worker  and  the  organization.  For  the  worker,  a  good  match  in 
terms  of  interests  can  result  in  higher  job  satisfaction.  For  the  organization,  more  satisfied 
workers  tend  to  work  harder  and  are  less  likely  to  attrit. 

The  Work  Preferences  Survey  was  designed  to  assess  potential  recruits’  preferences  for 
these  six  types  of  occupations.  Respondents  are  asked  to  rate  their  interest  in  various  work 
activities,  work  environments,  and  learning  opportunities  that  reflect  these  occupations.  The 
Army  can  then  compare  the  interests  of  each  respondent  to  the  interests  the  Army  supports  to 
assess  the  match  between  the  two.  The  current  version  of  the  Work  Preferences  Survey  has  75 
items  and  takes  about  20  minutes  to  complete.  Figure  5  shows  the  instructions  for  the  instrument 
and  three  items  that  are  similar  to  those  that  appear  on  the  actual  survey. 


Instructions:  The  purpose  of  this  survey  is  to  find  out  what  your  ideal  job  would  be  like. 
Please  read  each  statement.  Then,  using  the  scale  below,  indicate  how  important  each 
statement  is  to  your  ideal  job. 


1 

2 

3 

4 

5 

Extremely 
unimportant  to 
have  in  my  ideal 
job 

Not  important  to 
have  in  my  ideal 
job 

Not  important  or 
unimportant  to 
have  in  my  ideal 
job 

Important  to  have 
in  my  ideal  job 

Extremely 
important  to  have 
in  my  ideal  job 

Sample  Statements 

1.  A  job  that  requires  me  to  repair  broken  machinery. 

2.  A  job  in  which  I  work  by  myself  most  of  day. 

3.  A  job  in  which  I  can  leam  more  about  running  a  business. 


Figure  5.  Sample  Work  Preferences  Survey  items. 
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Career  Exploration  Program  (CEP)  Interest  Inventory 


The  Defense  Manpower  Data  Center  (DMDC)  maintains  an  instrument  to  help  high 
school  students  identify  their  career  interests.  The  recently  revised  version  of  this  tool,  the  Career 
Exploration  Program  (CEP)  Interest  Inventory,  is  similar  to  the  Select21 -developed  Work 
Preferences  Survey  (WPS)  but  was  designed  for  counseling  rather  than  for  selection  and 
classification  decision-making1. 

Both  the  WPS  and  CEP  Interest  Inventory  measure  an  individual’s  vocational  interest 
needs.  We  have  included  the  CEP  in  our  research  because  it  is  highly  useful  for  measuring 
occupational  interests  for  purposes  of  career  counseling  and  may  also  prove  useful  for  selection 
and  classification. 

On  the  CEP  Interest  Inventory,  respondents  are  asked  to  rate  their  interest  in  various 
activities  that  reflect  these  occupations.  The  Army  can  then  compare  the  interests  of  each 
respondent  to  the  interests  the  Army  supports  to  assess  the  match  between  the  two.  The  inventory 
includes  93  items  and  takes  about  15  minutes  to  administer.  Figure  6  shows  abbreviated 
instructions  for  the  instrument  and  three  sample  items. 


Instructions:  For  each  of  the  activities  in  the  inventory,  ask  yourself  if  you  would  like 
or  dislike  doing  that  activity.  In  answering,  don’t  be  concerned  about  how  well  you 
would  do  any  activity  or  whether  you  have  the  experience  or  training  to  do  it.  Just  think 
about  how  much  you  would  like  or  dislike  doing  the  activity  and  select  an  answer  from 
the  following  responses: 


1 

2 

3 

Like 

I  would  like  to  do  this  activity 

Indifferent 

I  don’t  care  one  way  or  the  other 

Dislike 

I  would  not  like  to  do  this  activity 

Sample  Statements 


1 .  Argue  a  case  in  court. 

2.  Fly  an  airplane. 

3.  Help  raise  money  for  a  charity. 

Figure  6.  Sample  Career  Exploration  Survey  Interest  Inventory  items. 


1  Department  of  Defense  (1999).  Technical  Manual  for  the  ASVAB  18/19  career  exploration 
program  (DOD  1304.1 2-L-AST-TS).  North  Chicago,  IL:  USMEPCOM. 
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Expectations  Measures 


The  last  three  experimental  measures  are  based  on  the  premise  that  not  all  applicants  have 
accurate  beliefs  about  what  it  is  like  to  be  in  the  Army,  and  that  inaccurate  beliefs  can  lead  to 
undesirable  outcomes.  Specifically,  past  research  suggests  that  people  who  have  accurate  beliefs 
about  the  organization  to  which  they  are  applying  tend  to  be  more  satisfied  and  less  likely  to 
attrit  than  those  with  less  accurate  beliefs.  Each  of  the  three  instruments — Army  Beliefs  Survey, 
Pre-Service  Expectations  Survey,  and  Army  Work  Knowledge  Survey — was  based  on  content 
from  previously  described  measures.  In  each  of  these  measures,  respondents  are  asked  to  indicate 
the  extent  to  which  each  statement  is  descriptive  of  the  Army  environment.  Responses  are 
compared  to  the  answers  given  by  a  sample  of  experienced  NCOs.  Because  they  concerned  pre¬ 
service  phenomena,  they  are  not  appropriate  for  administration  to  people  already  in  the  Army 
and  therefore  are  not  be  included  in  the  Select21  concurrent  validation  data  collection. 

Army  Beliefs  Survey  (ABS) 

The  Army  Beliefs  Survey  (ABS)  assesses  respondents’  knowledge  about  what  work  in  the 
Army  involves.  The  content  of  the  ABS  parallels  the  content  of  the  Work  Values  Inventory,  except  it 
has  a  couple  extra  items  for  a  total  of  30.  The  survey  takes  about  10  minutes  to  complete. 

Figure  7  shows  examples  of  what  items  on  this  survey  look  like.  When  a  respondent 
completes  this  survey,  the  Army  can  assess  the  similarity  of  respondent’s  beliefs  with  the  reality 
of  Army  life  for  first-term  Soldiers. 


Instructions:  Using  the  rating  scale  below,  indicate  which  category  you  believe  best 
describes  each  statement. 


Few  Will  Experience 

Some  Will  Experience 

Most  Will  Experience 

This  statement  describes  an 
experience  few  Soldiers  will 
have  during  their  first 
enlistment. 

This  statement  describes  an 
experience  some,  but  not 
most  Soldiers  will  have 
during  their  first  enlistment. 

This  statement  describes  an 
experience  must  Soldiers  will 
have  during  their  first 
enlistment. 

Sample  Statements 

1 .  Soldiers  will  improve  their  physical  fitness. 

2.  Soldiers  will  make  decisions  on  their  own. 


Figure  7.  Sample  Army  Beliefs  Survey  items. 
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Pre-Service  Expectations  Survey  (PSES) 

The  two  interest  measures  described  previously  (the  Work  Preferences  Survey  and  CEP 
Interest  Inventory)  assess  the  respondents’  degree  of  interest  in  each  of  six  occupational  areas  (e.g., 
Artistic,  Social).  In  contrast,  the  Pre-Service  Expectations  Survey  (PSES)  assesses  the  extent  to 
which  respondents  believe  the  Army  will  provide  work  activities  and  training  opportunities 
associated  with  each  area.  The  Army  can  then  determine  how  closely  respondents’  expectations 
match  the  realities  of  Army  work. 

Individuals  completing  the  PSES  are  asked  to  read  a  brief  description  of  each  of  the  six 
work  settings  listed  above  and  then  rate  the  extent  to  which  they  think  each  type  describes  the 
Army  work  environment.  The  instrument  consists  of  24  statements  (four  items  for  each  type  of 
work  setting)  that  respondents  rate  on  a  5-point  agreement  scale.  The  PSES  takes  about  20 
minutes  to  administer. 


Army  Work  Knowledge  Survey  (AWKS) 

The  Army  Work  Knowledge  Survey  (AWKS)  lists  the  same  16  statements  that  appear  on 
the  Work  Suitability  Inventory.  Respondents  indicate  the  extent  to  which  they  agree  with  each 
statement  using  a  5-point  scale.  Following  is  a  sample  AWKS  statement: 

Work  as  a  first  term  Soldier  in  the  Army  often  requires  showing  a 
cooperative  and  friendly  attitude  towards  others  I  dislike  or  disagree  with. 

The  AWKS  takes  about  5  minutes  to  administer. 


15 


Overview  of  the  Instrument  Development  Process 


A  fundamental  goal  of  the  Select21  instrument  development  process  was  to  generate 
experimental  selection  and  classification  measures  that  will  (a)  predict  performance  for  entry- 
level  Soldiers  in  the  Future  Force  and  (b)  add  incremental  validity  over  the  current  system  as 
embodied  by  the  Armed  Services  Aptitude  Battery  (ASVAB).  The  development  process  took 
about  2  years  and  involved  five  major  stages,  as  outlined  below. 

Job  Analysis  -  Identification  of  Pre-Enlistment  Knowledges,  Skills,  and  Attributes 
(KSAs).  The  Select21  job  analysis  team  reviewed  military  and  civilian  research  literature  to  form 
a  comprehensive  master  list  of  48  human  attributes  that  have  been  identified  and  measured 
reliably  in  the  major  domains  of  human  performance.  The  master  list  was  reviewed  by 
psychologists.  In  turn,  an  Army  subject  matter  expert  panel  and  psychologists  familiar  with 
Army  MOS  rated  the  importance  of  the  KSAs  for  future  performance.  Those  ratings  identified 
the  KSAs  that  are  most  likely  to  be  needed  in  the  future.  The  results  of  this  job  analysis 
procedure  are  detailed  in  the  second  Product  Report  in  this  series:  Future  Army-Wide  Soldier 
Performance  Requirements. 

The  Measurement  Plan.  Select21  ’s  objective  was  to  develop  new  measures  that  would 
cover  the  important  KSAs  as  comprehensively  and  efficiently  as  possible  within  practical 
constraints  on  administrative  feasibility.  The  job  analysis  defined  the  KSAs  that  needed  to  be 
measured,  and  we  focused  on  the  KSAs  not  already  covered  well  by  the  ASVAB.  We  also 
omitted  physical  ability  KSAs  such  as  strength  or  endurance  because  these  would  have  required 
physical  testing,  which  is  outside  of  ARI’s  mission.  Based  on  prior  research,  Select21 
researchers  identified  selection  and  classification  instruments  that  were  feasible  to  develop  and 
likely  to  measure  the  target  psychological  KSAs  reliably. 

Instrument  Development:  The  different  selection  and  classification  instruments  had 
varying  developmental  needs.  The  development  of  some  instruments  required  extensive  input 
from  Army  personnel,  while  others  required  considerably  less.  For  example,  development  of  the 
PSJT  involved  numerous  workshops  with  Soldiers  in  training,  drill  sergeants,  and  training 
instructors.  The  psychomotor  tests  which  were  adapted  from  prior  Army  research  were 
reprogrammed  with  little  need  for  input  from  the  field.  Each  instrument  was  developed  in 
accordance  with  its  particular  development  requirements. 

Pilot  Testing  and  Faking  Research:  Nearly  2,000  new  recruits  in  Army  reception 
battalions  participated  in  either  pilot  testing  or  a  “faking”  research  effort.  The  purpose  of  pilot 
testing  was  to  administer  early  versions  of  the  instruments  and  collect  preliminary  information 
about  how  they  were  working.  In  the  faking  research  data  collection,  we  administered  predictors 
potentially  subject  to  response  distortion  or  coaching  under  various  conditions  (e.g.,  respond 
honestly,  make  yourself  look  good  but  try  not  to  get  caught)  to  get  an  understanding  of  how  they 
might  function  in  an  operational  setting.  Although  the  Army  would  not  condone  coaching 
applicants  how  to  look  good  on  the  tests,  self-report  measures  such  as  these  are  inherently 
vulnerable  when  used  for  high  stakes  decision-making. 


16 


Field  Testing:  The  purpose  of  the  field  test  was  to  finalize  instrument  content  and 
administration  procedures  in  preparation  for  the  concurrent  validation  planned  for  2005.  A  total 
<  of  690  new  recruits  participated  in  the  field  test. 

„  Validation  Plans 

The  experimental  predictors  and  job  performance  criteria  (described  in  a  companion 
report  in  this  series,  Soldier  Job  Performance  Measurement  Tools)  are  being  administered  to 
Soldiers  in  a  “concurrent  validation”  effort  in  2005.  This  will  be  an  important  test  for  the 
predictors,  showing  which  are  related  to  job  performance  and  which  provide  predictive  validity 
over  and  above  that  which  can  be  achieved  with  ASVAB  alone. 

Some  of  the  experimental  predictors  also  show  promise  for  predicting  who  will  attrit 
from  the  Army  prior  to  completion  of  their  first  enlistment  term.  Data  were  collected  from  a 
fairly  large  number  of  new  recruits  as  part  of  the  instrument  development  process.  Researchers 
are  tracking  the  separation  status  of  these  Soldiers  to  help  determine  the  extent  to  which  some  of 
the  experimental  tests  predict  attrition  from  training  and  in-unit  attrition. 

Other  Applications 

The  Select21  predictor  tests  were  designed  for  use  in  initial  selection  and  classification 
decision-making.  It  is  certainly  the  case,  however,  that  some  of  the  measures  could  be 
administered  immediately  post-enlistment  (e.g.,  for  post-enlistment  classification  or  counseling). 

Many  of  the  experimental  predictors  developed  in  this  project  are  generic  enough  to  be 
potentially  usable  by  all  the  U.S.  Armed  Services.  Indeed,  just  as  with  the  ASVAB,  there  are 
efficiencies  to  be  gained  by  using  the  same  tests  across  services.  Accordingly,  information  about 
these  tests  has  been  shared  in  the  joint  service  community  and  this  dialogue  will  continue  as  we 
learn  more  about  how  they  function  in  the  Army  setting. 
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Who  is  involved  in  this  work? 


Select21  research  and  products  will  ultimately  result  in  recommendations  to  the  Assistant 
Secretary  of  Defense  for  Manpower  and  Reserve  Affairs.  The  offices  sponsoring  this  project  are 
the  Army  G-l  and  the  Training  and  Doctrine  Command  (TRADOC).  The  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  is  conducting  the  research,  largely  through 
work  performed  under  contract  to  the  Human  Resources  Research  Organization  (HumRRO). 

How  to  contact  us  and  for  more  information 

Your  reactions  and  suggestions  for  these  Product  Reports  are  important  and  we  solicit 
input  and  participation  in  the  Select21  project.  We  also  will  try  to  answer  any  questions  you 
might  have.  If  you  have  any  input  or  want  to  leam  more  about  Select21  and  its  current  status, 
please  contact: 

Dr.  Trueman  Tremble 

Selection  and  Assignment  Research  Unit 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  Highway 
Arlington,  VA  22202-3926 

Commercial  703-602-7765  (or  DSN  332-7765) 
e-mail  trueman.tremble@hqda.army.mil 


